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ABSTRACT 

A Study is being conducted to evaluate and revise, an 
innovative science program which was field tested in three schools 
representative of urban, rural, and suburban populations. The 
objectives of the study are to^ assess the program«s actual 
performance of its stated objectives and to inform the program 
developer of Inherent strengths and weaknesses, A phase of the 
evaluation design is the eKamlnation of the Placement Tests/ These 
were administered on a prepost basis for analysis. Results to date 
have been utilized in program revision and have provided evidence to 
support the need for formative evaluation of innovative instructional 
systems, (For related document, see TM 002 511.) (Author) 
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A FORMTIVE EVALUATION OF INDIVIDUAH2EP SCI EN^r 
AN INNOVATira INSTRUCTIONAL SYSTEM 



INTRODUCTIQN 

No matter how effective an innovatlva instructional system 
may purport to be. Its ultimata impact on education ±b determined 
by how appropriately and how widely it Is implemented in the 
classroom^ According to ^ler (1966), concurrent with the planning 
of an instructional aystem is the development of a means of 
assessing whether the aystem achieves its stated objectives. A 
critical evaluation of any Inatructlonal syitem is mandatory relative 
to the total developMntal process. Such an avaiuatlon will provide 
evidence concerning the appropriatenass of the content and processes 
of the program. 

In meeting the need for fomative evaluation in currlcur™ 
development j 'Research for Better Schools of Philadelphia, Pennsylvania 
Is conduetlng an extensive evaluation of an innovative science 
curriculimi throughout all levels of Its development. The results 
of this formative evaluation serve as the basis for altering the 
nature of the program in its developMntal stages* 

The program, Individualized Science ^ is being developed at 
the Learning Reiearch & Development Center at the University of \ 
Pittsburgh. : ae Imperial International Learning Corporation of 
Kankakee, Illinois^ has contracted the cdmTOrclal p^llcatlon of ' 
the program. RBS plays the unique k > of liaiion with developers, 
publisher, and the field test and demonitratldn schools while 



serving as the evaluator of the program. The Individualized Scien 



ce 



program Is basically a multi-media program directed toward the >' 
realization of a set of specific goals. 

FURPOSE OF TIIE INVESTIGATION 

It was the purpose of this paper to provide a brief description 
of the totsl formative evaluation effort extanded by ResBarch for 
Better Schools, Inc. relative to the Innovative instructional system. 
Indi vidualized Science . A more detailed description of one aspect 
of this evaluation effort has also been IneludBd in the paper. This 
.involved a detailed analysis of the unit placement tests for the first 
two levels of the program. 

DEFINITIONS 



Formative Evaluation . Scriven (1967) defined formative 
evaluation as the evaluation of educational programs still in some 
stale of development. The product of such an evaluation effort is 
expected to be an improved instructional program. ' 

INNOVATIVE INSTRUCTIONAL SYSTEM . Schutz ; (1968) refers to 
the effactlveness of instruction which is multiple mediated, 
individually paced, managed against objectives, and computer based. 
At the present stage of development. Individualized Science satiBfies 
to some degree the listed criteria and has thus been defined as 
Innovative. 

CONCEPTUAL FRAMEWORK 

In order to avoid a lengthy discussion eorieemlng the rationale 
of the formative evaluation effort suffice it to say that every 



accempt was made to adhere to the statement by Stake raiarding 
formative evaluation, "To be fully understood, the Bducatlonal 
program must be fully degcribad and fully judged," Add to this 
philosophy the limiting factors imposed by the legal contractual 
arrangement among the three concerned parties, RBS , LRDC, and 
Imperial, and the following sar of formative evaluation strategies 
emerged, 

EVALUATION STRATEGIES 

Through a consensus building process involving the developer, eval- 
uator, and publisher the following procedures ware determined and 
implementad In the collection of data. Formal reports daacrlbing ' 
the evaluation of each unit of the program wera published and 
distributed to the devalopBr and publisher on a scheduled basis. 
Following a review of the reports a meeting to be attended by 
representatives of the three parties is scheduled to determine the 
reasonableness of the evaluation findings for the final connnercial 
edition of the program, A sample unit report appears as Appendix A. 
The data CQllection procedures are described in the following outline ; 



PREPARATION OF UNIT REPORTS 
INDIVIDUALIZED SCIENCE 
FORJIATIVE EVALUATION 



Data from the three field test schools are continually 
forwarded to RBS. These data include for each unit (1) all pupil 
planning shests, (2) all pupil check-up lassons, and (3) all pupil 
placetnant tests (adminls tared on a pre and post basis). 

1. INITIAL ORGANIZATfON - DATA ANALYSIS ' 

a. Planning sheats are reviewed for all pupils for the 
specific report to be completed | clerks compile the data 
into tables ff^r review and reference, 

b. . Placenent tests administered on a pre-poat basis are 

analyzed, 

(1) Individual lessons prescrlbad from both the pre and 
post placement test results are identified. Appropriate 
recommendations are suigested for those lessons which 
apparently are not achieving their stated objectives, 

(2) The paper and pencil and the audio cassette presentation 
of the test is scrutinized by an RBS Test and 
Measurement Specialist, •Reconanendations are delineated, 

(3) An item analysis Is performed on each test. The 
results determine the reliability of the Instrument 
and the difficulty of each item. Subsequent changes 
are reconmended, ^ 

c. Check-ups (built-in unit diagnostic instruments) are 

reviewed In ordar to provide evidence for incomplete learnlnB 
experiences and/or poor test items. . » 



Problem sheets which contain comments and concerns are 
completed by teachers and reviewed, sumnarlzed, and 
interpreted as reconmendatlons. 
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e. Teachers Manuals which include written coments and 
criticisms are collected from each teacher and 
reviewed, Comments are compiled and aummarized. 

f. Observation data from regularly scheduled school visitations 
are compiled md syntheslzad, resulting in recoimnendations . 

g. Meef.lnga are conducted with all teachers at each of the 
thrtt achools prior to writing the initial draft. 

All comments are recorded; many problems are discussed 
and probable solutions derived. 

h. All lessons and activities are carefully reviewed for 
appropriateness and feasibility (simulated role play). 

(1) Each individual taped lesson, student activity, 
and directed group activity is experienced. The 
lessons (taped) are listened to twice; once for 
general approach, a second tlma for spaclfic directions. 

2. INITIAL CRAFT - WRITTIN AND TYPED 

a. The first draft is written and typed. 

b. The order of the reports generally includes' 
(1) Introduction i ■ 

. (2) General recommendations covering broad concams 

(3) Specific raconmiendations for each lesson and actimty 
followxtig the order of the lessons in the unit. 

3. EDITING ' ■ 

a. The first draft is submitted to interested parties for 
comment and editing. 

4. REWRITE - PINAL TYPE 

The length of time required to complete, the stated process 
of preparing a unit report has variftd according to the length of" 
the unit. An absolute minimum of two weeks for. a small unit Is 
required. Some units have .required three man months of intense effort 
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The remainder of the paper describes In detail the anaylsis 
of the specific data collected for the Placetnant Tests. 

PROCEEDURES 

mm Placemant Test is an inherent part of Individualized 
^g^sttce- The purpose of the Placement Test is to assess pupil 
knowledge of the scianca content in each unit. Lessons in the unit 
are prescribed according to individual parformance on the test. 

As one phase of the total formative evaluation of the 
Individual ized Science program, RBS conducted a study during the 
field tesclng of the Initial units of the program in the 1971-72 
school year. For purposes of the study, tha Placement Tasts were 
administered on a pre-post basis, thus employed as an evaluation 
instrument as well as a dlagnos-tlc tool of the program. The data 
were provided by three field test schools representativa of rural, 
urban, and suburban populations. 

The parposes of the Investigation were (1) to evaluate tha 
quality and effectiveness of each unit Placement Test In Levels A 
and B (grades 1 and 2) and (2) to establish a performance Indicator 
for the formative evaluation of each Individual unit. 

DESIGN 

An item analysis was performed on the PlacsnBnt Test 
results for each unit in order to examine specific items and to 
ascertain test reliabilities. Tests were necessarily divided into 
independent items for the analysis. A summary of this analysis is 
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presented in Table I. \ ' - 1 

In addition to the item analysiB, a pre-pogt comparison was 
made as a partial evaluation of transactions. A correlated t-test 
was used with the results of each test. It should be noted in 
Table ri that the decrease in sample size was attributed .to a lack 
of post test data return from all schools. The numbers reported are 
from the suburban and urban samples as designated. 

In addition to the statistical analysis an informal test 
format examination and content evaluation by RBS staff and eonsultants 
was performed in both the audio'-cassef te presentation and the paper 
and pencil answer sheet. 



TABLE I 
ITEM ANALYSIS SIMIARY 



TBBt No. of Itrnm Alpha 



Simpsori 21 ,703 (N^381) 

Galileo 25 .842 (N^390) 

Michelson 6 ,554 (N^366) 

Burbank 32 ,800 CN^213) 

Hooke 13 ,850 (N^lll) 

Curi^ 27 ,613 (N^65) 



TABLE XI 
PRE-POST COMPARISONS 



Simpson (N<1Q) 
Galileo CN^87) 

Michelson (N^71) 

Burbank (N^24) 
Hooka (N^12) 

Curie (data unavailable) 



t ^ 17,25^ Csuburban) 

t ^ (urban i N^55) 

t ^ 2,48* (suburban : N^32) 

6,40* (urban t N^33) 

t ^ 8,21^ (iuburban i N^38) 

2.71* (suburban ) 

t ^ 6,51* (suburban) 



Significant at the . 05 level , 
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RESULTS 

The following general results of the formative evaluation 
of the Placement Testing within the program were observedj 

1, significant differences were noted relative to the 
correlated t-test between the Placement Test administered 
as a preteBt and the iame Placemafit Tast administered 

as a posttest for each unit investigated (See Table II) - 

2. item analysis utilizing the initial Placement Test res uits 
provided evidence for the revision of specific items 

of the tea t ; 

. 3. the reliability of the instruments was ascertained providins 
additional evidence for deslrabla changes in the test 
(See Table I) ; 

4. format and audio-tape Inconsistancies were discovered 
relative to the organization and visual presentation 
of the test items. 

It was concluded from these results that certain Placement 
Test itemb must be revised, added, and/or daleted| that individual 
unit lessons must be revised because of the evidence indicating lack 
of significant achievement ; and that format changes must be introduced 
in order to eliminate confusion resulting from Inadequate visual 
and tape presentations. 

The results of the evaluation of the Placement Testing component 
of the total Individualizad Salenaa nrnor^w reflective of the total 
formative evaluation desl|n presently being implemented by Research ■ 
for Better Schools, Inc. 



CONCLUSIONS AND RECOIMNDATIONS 



Of Che six Placement Tests, mehelson and Curie eidiibit 

■ . : v.. ■ / . , ■ . - . ■ ' '\ ' ■ ' 

questionabla Laliabillties. It is recommended that the Mchalson 
test be lengthened and that the items and concepts In the Curie 
test and unit be further examined and revised^.in order to increse 
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. their reliabilities. 

The significant differences observrd in each prd-post comparison 
hold positive signs towards the attainmenc of the prcg^^ 
The use of control schools in the future will lead to more ganarallzable 
conclusions relative to transactions and achlevemant. 
nDUCATIONAL Ifrff ORTANCE • , 

When the development of an Instructional lys tern hag been 
completed and Its effectiveness demonstrated, its educational Implications 
are considerable, ■ _ 

The RBS study has demonstrated a cooperatlva professional 
^ effort in the evaluation and developmsnt of an innovative educational 
program. It exemplifies the practical and unique relation^^ 
caranunlcation which can and should exist between' educational research, 
curriculum development , commercial production, and the "grass roots" 
practitioner. Such a relationship is invaiuable to our goal of quality 
education and should be spread throughout our educational syi tern. 

In addition, the study has exhibited the purpose and importance 
of formative evaiuation in curriculum development. It has showi the . 
direct ralationshlp between objectives and evaluation -that is, 
evaluation is essentially a process of determining to what extent 
"^"^^^ specified are heing realized by the Instructional 

program. 
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